The Big Blog

a_hylian_human commented Wednesday, Dec 29

So all of this could have been prevented if Skype's auto update actually worked. Just a quick note here: every update I've ever gotten for my Skype client was through a manual upgrade check.

fasesky commented Wednesday, Dec 29

First of all I would like to congratulate with you Mr. Rabbe, because of your detailed explanation.
Skype is a very great service and every day I rely on it, for both work and personal use.
Every person can do an error, but there aren't many people that are willing to admit theirs guilties.

itcodemonkey commented Wednesday, Dec 29

I wasn't impacted too much by the outage. However, Leo at twit.tv was for a couple of his shows. It proved how much Skype is needed as their other options were limited and not that great. Everything they do with Skype is fantastic and when they had to switch to other methods, it just wasn't the same. The quality was way down.

On one hand, the outage was bad for a lot of people who couldn't use it.. On the other hand, this will help Skype identify any weak points and patch those up.

gerard.roelands commented Wednesday, Dec 29

Thanks for your explanation of all troubles since the dsipruption as well the way you tackled the disaease.
Fully understandable for me, because although I'm retired (almost 65) and formerly worked in an hospital (Hospital Information Systems) I "enjoyed" your explantion how jou tackled this major disease.
As the french say "CHAPEAU".

Personally my skype activities never were down in my location.

With deep respect I wish you lots of succes in the future and a happy new year.

king regards
Gerard Roelands
groelands@zonnet.nl

santeld commented Wednesday, Dec 29

Lars,

Having worked for the likes of AT&T, it is fantastic that you've taken time to explain the issue as you have, along with the plan to insure this will not happen again. It's not often candor such as yours is expressed by a CIO. It demostrates a maturity and confidence in your team and yourself. Please keep up the great work!

BTW, I've just signed up for your service for this very reason.

telesquirrel commented Wednesday, Dec 29

I appreciate how you have handled this incident. You've shown a Skypish kind of customer care we all appreciate in your actions and communications.

mrezak2 commented Wednesday, Dec 29

Hi
Good to hear that everything go to normal again! I'm not testing new version (156) yet but I do not like any of v5 of skype! (Due to some old features leak) I prefer to use v4.2.0.187 and any time I'm reinstalling v5 to test it on different ISPs at Iran, I had to get back to old v4 due to faster connecting to server and more stability.

Another thing, I hope you guys can find a way for us at Iran to bypass some filtering on skype client so we can connect to skype without using any VPN. most ISPs block skype at here!!!!

I love Skype more than others messengers for 3 thing: 1) easy to use for my family 2) really unique and great sound Quality 3) Desktop Sharing which help me to show my families to do somethings at their systems via this great feature.

Love you and I hope you have great holidays beside your families and more greater and successful business days at new year!

Best Regards
MRK

olness commented Wednesday, Dec 29

The automatic software update contributed, in part, to the Skype
outage last week.

When I used Skype last week, the program was automatically upgraded
to the "buggy" version while I sat and was forced to wait.
If I could have chosen to perform the upgrade on my own time
schedule, I would have done the upgrade in the future so I did
not have to sit and wait for the update
when I wanted to make a phone call; thus, my computer would have been
available as a functioning super-node (running the bug-free older code)
during last week's outage.

oldmrbill commented Wednesday, Dec 29

Kudos to CIO Lars Rabbe for a straight-forward explanation of the outage. No smoke and mirrors, just the facts. How refreshing!

redpanda88 commented Wednesday, Dec 29

Thank you Skype customer services for acknowledging the problem, taking responsibility for it (although technology failure is impossible to avoid completely), and giving customers a credit voucher. It shows you remember who keeps you in business; a lesson some big companies have yet to learn!

markskessler commented Wednesday, Dec 29

And after that lengthy explanation which probably 99.99% of people will not understand (super-nodes, p2p clouds, mega-nodes - really???) you are giving your long time, loyal customers a whole $1 credit to compensate us for the outage. Wow - that's incredible!!!! Merry Chrsitmas indeed. Reminds me of the Mortimer brothers from Trading Places - one whole dollar.

basiclife commented Wednesday, Dec 29

Thanks for the information - very informative and with hindsight, an obvious problem (isn't that always the way). Congratulations to your engineers for identifying and solving this problem as quickly as they did. I work as a developer and understand how much of a pain this must've been.

I appreciate that Skype is willing to acknowledge and explain its' problems rather than try to hide them - You've won some good will from me for owning up and explaining.

sangdelan commented Thursday, Dec 30

What if someone doesn't want to update the release of Skype? I am perfectly happy with my old one and I don't see any reason why I should change it. The few times that I tried, I always went back to the old one, as the new releases I did not like and gave trobles to my PC.
Last week I saw that often there were attempt to "auto-instal" a new version of the programme. I was quick enough to avoid it.... Will I be able to avoid it also in the future?
I hope so.
Regards.

ultra-fine commented Thursday, Dec 30

Having read the entire explanation I still cannot get how the Skype network infrastructure works. Are supernodes - just usual PCs running Skype? Does that mean that you offload some of your core networking services onto users?

Can you draw a little chart/scheme how everything works?

rickardmelkorhedman commented Thursday, Dec 30

After i read that message i was very pleased. Hope all goes well with the plans for further stability.

hendrakieran commented Thursday, Dec 30

Skype is super Awesome! good to know, thanks for sharing this.

melaperson commented Thursday, Dec 30

I don't understand why your email to me suggests that I will get 30 minutes worldwide but when I put in the code that you sent me, it says "Your voucher or prepaid card was succesfully redeemed, and we are now activating the following services:
$1.00 Skype Telephony Voucher for $1.00" Which is true? I call Ecuador. $1.00 won't cover ten minutes.

nyusoft.tech commented Thursday, Dec 30

Dear Mr. Rabbe,

We are proud to use Skype and I am using Skype for more than 1.5 years for my software business. And this is the first time I got problem with Skype. But the one who works hard has to give exams. So it was your exam time, and you passed it with flying colors. Dont be sorry or think that you fell short to users' expectations. You are the best.

Thanks for your services and great support.

Regards,
Nirav Vora

kzinvogon commented Thursday, Dec 30

Well done! What a refreshing approach to service. that's why your company is highly respected and will continue to be so. Without skype my business would still be paying rip off teleco bills. Thanks to skype I have emplyees on every continent and daily conference calls and for sure my conventional voip (non peer to peer) is always out performed by Skype.

robsoles commented Thursday, Dec 30

Hey! Just a couple of things...

(1) Where my dollar be at? I read a few people complaining they got a dollar for their troubles and my Skype was as inaccessible as everybody elses in that period so, ahem, cough up

(2) I see the updates are still coming thick and fast for Skype (Beta) 2.1.0.81 for Linux huh?

I'm using sarcasm for #2 there, I really do want the dollar in #1

stevie_chambers commented Thursday, Dec 30

Perhaps it's time for Skype to host some supernodes, for paying customers? Piggybacking on other clients might be "scale for free" but it's unreliable (as we've seen) and I wonder what an evil person could do if they pretend to be a supernode, or subvert a supernode?

andres_crespo commented Thursday, Dec 30

We can all run into unexpected problems from time to time. Understandable. HOWEVER, do not say you are sorry, if you do not mean it. Sending a $1 (yes, I do mean ONE pathetic dollar) is adding insult to injury as for those that are regular Skype users paid over the odds to use alternatives during the outage. I only found out about the outage after hours trying to connect. Perhaps an email from Skype as soon as they found out would have gone a long way to reduce anxieties. Skype chose not to inform users until problem was under control (too late, we already knew it was sorted!). Perhaps apart from reviewing their testing procedures (which proved to be insufficient) they should MAINLY review their emergency communications strategy (if they have one!).

stylinexpat commented Thursday, Dec 30

From what I understood is that they were to credit everyone 30 minutes. So what turned out to be giving people 30 minutes of free calling to landlines anywhere in the world was actually just a $1 credit!! WTF was that all about?? Why not just give the people 30 minutes credit of international calling then to where they have 30 minutes of credit towards all their international calls instead of this stupid $1 credit thing. If one was calling the middle-east that $1 would be gone in a few minutes time. Bottom line is Skype pulled a fast one by lying about the 30 minutes credit for international calls.

The voucher can be used to give you approximately 30 minutes of free calling to landlines anywhere in the world.

dylan.... commented Thursday, Dec 30

Lars, thanks for the detailed explanation; it is clear that the trigger, a buggy client, should be shot.

But your claim that the software is robustly tested does not wash. The 152 version also crashes on log-in for user names containing more than two consecutive periods ("..."). This is the kind of bug that should be spotted in routine test scripts, checking the user name base or at least a subset of it. I've had to go back to 4.2.0.187 AND turn off automatic updates, because a subsequent version I unwillingly installed did not fix this bug.

I'm afraid this is pathetic Lars, and no amount of explanation can correct shoddy processes. Hire someone who knows about testing and sort this out. In the meantime, I'm sticking to 4.2.

nils_vogels commented Thursday, Dec 30

Let's just say I wanna help out and dedicate resources to running a supernode, which will be online and on high bandwidth. How can I help?

marko7474 commented Thursday, Dec 30

Hi, thanks for the update. It was horrible because I had a conf call planned at the time.
However Skype remains one of my favorite apps.
Also one remark, I have so many people in my contact list and I don't need to see their photos, would be better is there would be an option to change the views.

thanks, marko

linuxmaven commented Thursday, Dec 30

I was impacted by this outage, but have not seen any offer from Skype to compensate. I do pay Skype monthly. Let's see --

jnevillerolfe commented Thursday, Dec 30

Thank you Mr Rabbe for explaining. Really appreciate the honesty, and good luck with China!

t.mattsson commented Thursday, Dec 30

I'm not going to comment on the problem itself -- I hardly noticed it -- but it's nice to see a fairly detailed description of what happened. And what's even nicer is how polite and encouraging the user comments are.

Maybe the Christmas spirit hasn't worn off yet.

sinketa commented Thursday, Dec 30

yes, it is kind of ridiculous to offer 1 dollar as compensation for lost services?? I appreciate excuse, but it cost us all money and we relied on Skype, so this compensation is rather insutling

josephspringer commented Thursday, Dec 30

Yes, I agree. It is sort of incredible to see such a detailed, thorough, and honest explanation of a company's technical issues. Thank you for the forthrightness.

Skype 4.x does an auto upgrade whether I like it or not. Skype 5.x audio is so choppy (both receive and sending) I cannot use it (Vista 64-bit), I have tried both regular and business. So I downgrade to version 4 which works great, but auto upgrade kills me every time. What to do?

jgodfrey33 commented Thursday, Dec 30

I am extremely impressed by this post. While the issues were unfortunate, I think Skype's response to them has been very constructive and transparent.

In the future will you more rapidly be able to roll out mega-supernodes? Have you looked at deploying instances on to a cloud service?

Have you considered a beta branch of Skype for the more technical crowd (e.g. Firefox nightly builds)? While you do offer betas it doesn't update to the latest test version (and you also introduce bugs between "beta" and "live").

aj.pearlcarroll commented Thursday, Dec 30

I completely agree with the other posters that while the down time was unfortunate, I think your efforts to fix it and your detailed explination were a very refreshing view of what is going on and what happened. Sure the outage was unfortunate and caused issues for people. I know I had difficulty moving information around during it, but without Skype I wouldn't be able to do what I wanted anyway and nobody else comes close to the transparency at the same level of service as Skype does. Thank you very much for your efforts.

AJ

tenay10i commented Thursday, Dec 30

I echo previous comments of appreciation regarding how you handled this incident, your detailed explanation, and the vouchers you sent to your customers. Your attitude and customer sense is dead on. I will tell my friends all about how pleased I am with Skype. Happy New Year!

nagyrakoi commented Thursday, Dec 30

There is something I do not understand about the super-node and client interdependence.
Can a user's client be a super-node? Can my PC e a super-node without my knowledge?
If yes, then what does it mean in degradation of performance on my PC?
Is this the cause of the irritating delay of about half a minute between calling a partner and the ringing tone?

jeremy.nederhoff commented Thursday, Dec 30

I'm curious to know if your development team reads any of the power systems research? They have been dealing with cascading failures for quite some time, and there may be some research done there that is useful to Skype's operations.

john.s.ryan commented Thursday, Dec 30

Mr. Rabbe:

This is a model example of how to respond to such a situation. By being sufficiently transparent about the details of the event, speaking plainly about what went wrong, and enumerating how you're addressing the problem, you've repaired and engendered trust.

Not only have I learned that Skype has a robust service (and is improving); I've learned that their CIO has integrity... in copious amounts.

Well done.

Regards,

John S. Ryan

ziggy2692 commented Thursday, Dec 30

Everyone,
I just wanted to say that i was offered (like everyone else i'm sure) a credit for the outage. I just wanted to say that i will decline to take a credit. I have been very happy with the service that i have been given by Skype over the years, and proudly pay the 50 dollars or so a year for the unlimited land line calling. The service is excellent, and with anything in life there will always be issues that arise from time to time. I find it troubling that i see as many complaints as about this as i do. For what most people get for free, i gladly will continue to pay for. Excellent product.
Thanks!

blakeyrat commented Thursday, Dec 30

Thanks for the explanation of the outage.

Isn't this the same fundamental cause of the outage a few years ago, in which Windows Update restarted a significant number of computers simultaneously, causing the Skype super-nodes to overload in the same way as described above? In that case, the original cause wasn't Skype clients crashing, but being shut down due to OS patching, but the rest of the explanation still fits.

In the future, you may want to have a super-node, when shutting down due to traffic, to signal the network to start up a cloud server at Amazon (or another provider) to replace itself-- that way you could save the cost of running your own super-nodes all the time, while still ensuring that enough super-nodes stay running during events like this.

dpezely commented Thursday, Dec 30

Guidelines used while developing the Internet Protocol stack, as conveyed by Dave Mills (a.k.a., Mr. NTP himself, creator of fuzzballs, original IP routers) to his university students:

1. You cannot anticipate all the faults.
2. All fault scenarios will happen at least once.
3. No one single strategy will work.
4. The system must be self-correcting.
5. But each correction must not increase vulnerability.
6. No system will always obey these rules.

Amendment:

* No system works correctly.

Acid test:

Would you trust your paycheck to the system?

Implementation Hint:

Use unique timeouts such that even in combination they become signatures of where problems may exist. Use a different prime number for each to aid in discovering the combination of compounded timeouts.

maitriquinn commented Thursday, Dec 30

Thank you for the transparency. As I didn't suffer during the outage, I won't redeem the outage voucher. Your high standards for service, functionality, vision and performance have kept me, for one, on board as a satisfied customer. Happy 2011 to you, and to all.

mark.cusumano commented Thursday, Dec 30

I have to say. I've dealt with many of the other phone companies both personally and professionally. Things happen in a large network. What separates the good from the great is honesty and communication. We have a term we call "phone magic" we use whenever we report a problem to a telco and they claim nothing is wrong but the problem mysteriously goes away later. Skype has not only been honest about the situation but has even taken measures to compensate users for the problem. Something other companies usually refuse to do unless you complain.

As for the Auto Update I'm not sure what the problem is there as I'm always notified that an update is available and install updates as soon as I'm aware of them. I think the problem is configuration and Skype should come with Auto-Updates fully turned on by default as a suggestion.

jrcmatthews commented Thursday, Dec 30

I remember Lars when he was at Fidelity Investments, he is one of the straightest shooters in the technology biz. I use Skype regularly and will continue to do so with confidence knowing that honest, hard-working and dedicated technology professionals such as Lars Rabbe are behind it!

Regards,
Jason Matthews

leslieptjeff commented Thursday, Dec 30

I too am just stunned at the generous $1 Skype credit - a 30 second call is about what it will cover -
I also received what looked like a phishing message when I logged on during the outage period; I blocked and reported it, and hope that no one on Skype was harmed ....

chris.dowsett commented Thursday, Dec 30

Skype - I really appreciate you taking ownership of the problem and trying to explain the situation. But a $1 voucher ...? Originally I read this and thought "This is a good company, they've owned up and they're making it right with a voucher for 30 minutes." Then I saw the voucher amount after I logged in and redeemed and ... well, honestly I think it's a pretty poor showing. I think it might have been better if you didn't offer the $1 and just left it at this blog and apology.

fleetwooda commented Thursday, Dec 30

Dear Lars
Read about the problems and well, this kind of thing happens, read also about some users may receive compensation, well your/the skype service in my view so just beyond fantastic. Therefore, if some form of compensation is to come my way PLEASE keep it just keep on providing a fantastic service and l am happy to put up with the odd disruption. Happy New Year in advance to all the Skype team, from top to the cleaner.

Best regards, fleetwooda

pim-veld commented Thursday, Dec 30

I like Skype and I like the openness about the problem. But I am afraid that it is not the whole explanation. I use the PC-PC and the Skype-Out facility. Unfortunately the later versions seem to be less stable. At the moment I use v. 5.0.0.156 and regularly encounter the infamous message "This program encountered a problem and had to close ..." It seems that most of the effort goes into adding more bells and whistles which most people don't use but decrease the stability of the program. Further I hate the loss of the possibility to "ring" the computer's internal speaker at incoming calls.

gh-lcl commented Friday, Dec 31

Yes, it is good that the CIO can be so frank, it's a shame the support staff don't share his candour and zest for problem solving. After the recent issues I had an idea that I literately had to spell out to the support staff and ask them to pass it on to the product team. She promised that they would reply to give me some confidence that my idea was being considered.... no reply yet.

Why can't users (businesses) set up their own private super nodes? You run Skype, perhaps in super-node only mode on your DMZ with some kind of filter to allow it to serve only your employees (perhaps by username or just a specific port). Then allow clients to add their own preferred super-nodes to their list. Thus ensuring your employees communication is not reliant on random peers whilst not having the burden of being a public super-node .

What do you think?

demagneto commented Friday, Dec 31

Thanks for the detailed explanation.
I have been in the software development business - so I am sympathetic to your frustration. More pathological testing seems to be in order for your upcoming releases.

mitcheg commented Friday, Dec 31

Dear Mr. Rabbe,

This response should serve as perfect example to all service providers on how to deal with public relations and outages. Open, honest and detailed. As an IT manager I can only imagine how intensely your operations staff worked to remediate this. Few people, so very few people have any idea of the extreme complexity of the Skype IT product mix, the scope of the virtual and real infrastructure and how much of it is outside of your direct control.

Everyone has 20/20 hindsight. The value for money ratio provided by Skype is world class. If, in the years of perfect service I have experienced, I must endure one single day of outage, in order that your company can unearth, understand and kill such a corner case threat, then so be it. You are pushing forward a wave on the boundaries of technology and we surf that wave with you.

Thanks for the wake up call to just how valuable your service is and good luck in the future.

Graham Mitchell

rnc000 commented Friday, Dec 31

Thanks for the info. Skype is a great product. Keep up the great work!

james_melrose commented Friday, Dec 31

Wow! What an excellent description.

My personal opinion of Skype has gone up following this technical issue, simply on the way you handled it! I think most others would agree.

Funny how a disaster can actually be mitigated by good customer service and an open honest approach. If only other companies acted similarly.

shazz9 commented Friday, Dec 31

Awesome thanks for the detailed explanation and compensation of 7 day credit.
Skype is and will always be my main mode for business communication remotely or office as well as for family connections.
Will continue to support such a excellent company..

Happy New Year Skype, see you next year & the next & the next.....

Regards
Shazz

zeldafan678 commented Friday, Dec 31

The humorous thing in all of this, is that if no one would of updated to 5.0 the problem would never of happened, stupid buggy 5.0 update, GO DIE.

I am very glad that I HATE AND DESPISE the new skype 5.0 so therefore I wasn't apart of the problem of the network going to down, due to their stupid useless and buggy 5.0 update.

OH that thing about automatic update, I choose to NOT update my skype to 5.0 and I won't until the make it work exactly like 4.0 does.

They sacrificed the mic settings somehow that makes them 10x more sensitive than necessary and it causes the audio to bounce around and raise and lower on it's own.

I do not have this problem on skype ver 4.2.0.187, it work as it should and how I've grown to love it.

I do not want to be forced to update to any client in the future.

abstemu commented Friday, Dec 31

Great attitude and approach from an Executive of a Service Provision organization. Straight talk - an alien trend in the marketplace!

I remain your customer!

isagiovanetti commented Friday, Dec 31

Thank you to take the time to explain the situation.... I use a Mac OS X and I'm a Skype user for at least 3 years now and I've to say that this is the first time I had problem with.
Thank you also for the voucher, I can understand some people frustration but although is only 1$ voucher (1$ for me but how many for you?) I think is a gesture of appreciation for your customer. Keep up to good work!
to all the Skype family Happy New Year!
isabella

scorrado100 commented Friday, Dec 31

Thank you for explaining what happened and for your honesty. It is extremely rare that businesses explain to paying customers what has happened and then offers credit, and as of today even an extension of 1-week's service. Usually there is no explanation and "what is lost is lost -- too bad for you (the paying customer)". This attitude absolutely annoys me. So thank you for your openness. You (Skype) are definitely above everyone else in my books.

jagmohan.nanaware commented Friday, Dec 31

Dear CIO,

Thanks for the explanation. A very nice gesture from skype to reach every customer. Most importantly considering everyone - even if they use free PC-2-PC - as customer is a great gesture!

regards,

Jagmohan

sam.whitsitt commented Friday, Dec 31

Hey folks -- I appreciate your explanation. I for one am surely NOT going to ask for any voucher from you folks. You provide an incredible service for which I am very grateful! Thanks so much!

giorgio.guastella commented Friday, Dec 31

First of all thanks to Skype for the services it offers!

... second Is better to send a mil with the word "sorry" that a mail with an empty voucher!!!

I wish you a better 2011!!!

julesk72 commented Friday, Dec 31

Thank you for your efforts to compensate and appologise. Greatly appreciated.

josenairobi commented Friday, Dec 31

Please don't autoupdate without permission from the user. Some of us have a data plan with limited downloads, or we just need our full bandwith at a particular time. Instead, show a message prompting us to update, and informing whether it's a critical update or a minor fix.

yvan.gauthier4 commented Friday, Dec 31

Your failure of December 22 is a sign of incompetence, your reply is pure bla bla and that the rate increase you have put up is absolutel outrageous. You do not help to keep clients with increases of 150% in my case from 7.85 for three months to 6.99 for one month. I won't renew and will surepy try and find alternatives. Not to mention your inability to serve well your clients using samsung galaxy under android .
Regards

Yvan Gauthier

emeraldrainbow commented Friday, Dec 31

I agree with the comment above by markskessler:
I really appreciate the long and thorough explanation, and I accept all the apologies, but a voucher for 1 dollar is just ridiculous. Just the apologies would have left a better impression.

luke.landis commented Friday, Dec 31

To the great people at Skype,

Everyone has an off-day, but not everyone is so forthcoming and honest. Thank you for your gracious emails, straightforwardness with the press, and technical detail on how Skype works. I am very happy to call myself a Skype subscriber and continue to recommend the service to my friends and acquaintances all over the world.

Again, thank you, and you freakin' rock!

Most Sincerely,
Luke Landis
Vancouver, Canada

cynthiacollett commented Friday, Dec 31

I am so very impressed with Skype and how you stepped up and acknowledged the problem, explained it to us (even though very few could actually understand it, I still appreciated the work you went through to give us answers), and offered us compensation beyond the amount of time some of us were without Skype. Your customer service and genuine concern for the quality of service we get harkens back to an era when people actually expected to be treated with respect.

The more I use Skype, the better I like it. I'm now considering having incoming calls as well as outgoing (land line) calls.

Happy New Year to all of you.

faktorl commented Friday, Dec 31

thank you, Lars, for the profound explanation. i run an online-based project and i'm constantly amazed at how you guys have been able to pull off such a great service as Skype. we cannot imagine our lives without Skype anymore! i wouldn't be able to work between two continents without it..
one day with interruption is not a problem - and i personally didn't experience any issues on that day.
happy New Year to you and all the team behind Skype, Liza

dani.tal commented Friday, Dec 31

I think your problem is Quality Management or in other words plain QA.
Nor overload and not version distribution.
Dani.Tal@cio-1.com call me for feedback.

ryanaslett commented Friday, Dec 31

Thanks for the explanation, and I appreciate the offer for some skype credit, however, several users, including myself have received the sorry for your trouble email with a blank code. I dont know if gmail stripped the code out, but nevertheless, it looks bad. Oh, and way to post a video about the new iphone video calling feature that you cant watch on an iPhone.

bill_mcgonigle commented Friday, Dec 31

I find it remarkable that your system is so large and successful that it's not possible to fully test and simulate it before release. We never covered that one in software engineering class. The life of the pioneer is always full of surprises.

mstifts commented Friday, Dec 31

If you wouldn't force updating to Skype 5 without even asking the user, the percentage of clients running the buggy version would probably have been low enough to not cause any harm.
Instead you're annoying all the people who like Skype 4 way better than Skype 5 in a rather rude way. (And as I don't use group video conferences, I won't update, because I would lose some essential features.)
If you should decide to completely block the Skype 4 clients one day, I will completely drop Skype, if the new version can't fulfill my personal needs, which only Skype 4 can do so far.

Also, your Email is pretty much indistinguishable from a Phishing attempt. It contains:
- External graphics
- Several links which don't point to the URL their link text is suggesting, but something cryptic instead, which seems to be used for email click tracking.
- Instructions to enter the Skype account name and password immediately following one of those seemingly fake links.

r.flood commented Friday, Dec 31

Anybody can have problems and have an outage. Its ok. The thing is that I received a mail explaining all this, and also saying that I had a credit to talk around half an hour. That I only had to put my user name, passw and the credit code which I never received. Im really surprised....

yeomantechnology commented Friday, Dec 31

Great explanation, but you forgot to make one recommendastion for users; always have a backup plan. It's surprising that many of the complaints posted on the web are from users that relied soley on Skype. The frameworks of modern technology (Internet, Distributed Computing, Open or Semi-open source software) are great but they all have the potential for failure, even Skype. It's a fact of life, organizations need to plan backups for everything they do. Bravo that you stepped up and acknowledged your mistake and are planning around it. However, you can't control the enitre web; outages will occur, users should plan accordingly.

lorsconsulting commented Friday, Dec 31

Thanks for the candid explanation. I appreciate the technical detail because it helps restore my confidence in Skype. I recognize that no software is perfect or bug free. As long as the crashes are extremely rare, I can accept them.

While it is good that you are reviewing your vulnerabilities, I would encourage you to also put emphasis on improving version 5. Group video calling is a valuable feature to add, but it has come at price. I used to be able to run 720x480 video from my DV camcorder connected my PC firewire input and/or S-Video capture card. But not anymore. These inputs no longer work with Skype! I now have to use a third party product to capture video, and then it will only do 320x240. I had abandoned alternatives to Skype because Skype was the best. However, if version 5 is not tweaked soon to be as good as version 4 was, I may find myself using one of the others again.

I will say that your audio quality still remains heads and shoulders above the competition, and is worth the $3/mo. cost over your 'free' competitor (which sounds like a cellphone call).

kley_moment commented Friday, Dec 31

Give us back those full installations and it'll be easier to be up to date. It is kind of boring to steal a real setup file from an online installation every time or to download it on every machine.
Also, you repel people from updating by adding features/making changes people don't want. I caught myself few times unwilling to update it even knowing how crucial it is to have up to date software. Just because previous version was faster logging in and had less animated stuff.
At some level of development a program like this would have separated it's transport layer from its GUI. But this will never happen for Skype because of it is closed.
Kley Moment,
Ukraine.

slvdogg commented Friday, Dec 31

Thank You very much Skype and Mr. Bates CEO of Skype for the very detailed explaination and also for extending my online number and giving the credit as you did.
The way and the timely manor in which you and your company handled the problem was "Top Notch Professionalism" at it's best !!! P2P, Nodes, Super/Mega Nodes or whatever, I'm not certain that any cell phone provider or landline company would come close to taking care of it's customers as you have done, especially with the amount of people who use your service and the timely way all of this was handled.
Thank You Again!!!

Best Wishes for 2011
david

fielious commented Friday, Dec 31

I would also like to thank all of the Skype team for explaining what happened. I was not affected personally by it. It take a great company to see a problem and explain why it happened. Not like my ISP who every-time something is going wrong they say that it was a fiber cut. I swear they have cut the same fiber line 100 times by now. I would think they would know where it is.

Thanks for your honesty,

Fielious

dcsutherland2084 commented Saturday, Jan 1

THANK YOU for the transparency.

The plan going forward looks good.

Other mission critical software providers should take note of your prudent measures and emulate them too.

bromia commented Saturday, Jan 1

Thank you for your explanation (most of which I will admit I didn't understand) and thank you for the extended subscription gift. But most of all, thank you for the WONDERFUL service you have provided me since 2005. Skype is absolutely priceless to me, I cannot believe how little I pay to stay in contact with my friends and family worldwide. Considering the vast number of customers you have (and how much FREE service you provide) it did not surprise me that the system was down for a day -- although I assumed that it just got overloaded for the holidays. Anyway, you guys are great. Keep up the good work! And don't fret over this little failure.

jeff7091 commented Saturday, Jan 1

You clearly have an architecture problem. Better client testing is a whitewash (but maybe it is all you can do at the moment?) If you had a resilient architecture, you could easily tolerate lower software quality. Fix it now, or suffer again. Mr Bates is familiar with routing protocols, and will be able to help you in this regard.

azninja commented Saturday, Jan 1

I wish all companies took after Skype. Now THIS is what I call customer service and it's exactly why I am a loyal customer.

matt06sg commented Saturday, Jan 1

Hi.

Happy New Year.

Thanks for the report. It is good that Skype took the right approach and provided customer with a detailed honest explanation.

Keep up the good work and hope such situation or similar will not happen again.

Thanks.

Matthew

peter.m.hentrich commented Saturday, Jan 1

Nearly every s/w co. has to deal with updates, bugs, different client versions etc. It is a massive task to cater for all these factors not to mention the millions of users. I can imagine the pressure upon the engineers to resolve a complex issue as fast as possible.

Due to the nature of P2P technology, it's not so easy to pinpoint the cause, so all you 'users' out there, don't be quick to point the finger!

Of course, not every person will be pleased by an outage but I believe that Skype has performed well both in it's technical rectification, news updates, this final debrief and compensation.

Keep up the good work. Your honesty and excellence are the signs of what makes a business successful.

Like 'santeld' and 'basiclife', I have worked in development and technical support for both private and govt sectors. It is rare to see such technical transparency and honesty, especially from the CEO.

'melaperson': Please read the fineprint. Skype says '30 minutes to a landline in some of our most popular countries'.

'ultra-fine': You are on the right track. P2P relies on decentralized servers, however, Skype will have their own servers for routing the calls and other . In a nutshell, the problem exists with the nature of a 'super-node', which is a universal directory of what users are online. If the 'phonebook' isn't working, then you can't call anyone. Research 'supernode' and P2P at Wikipedia. Drawing a chart etc of Skype technology could lead to divulging confidential information.

luisfalonso commented Saturday, Jan 1

Thanks for this detailed explanation... This is what i expect from a reputable company, and strengthens my confidence in you.

dgorzynska commented Saturday, Jan 1

Hi,

After couple of days of frustration of not being able to use skype i was glad to see the email and receive that small token of appreciation....only that the coupon code is NOT included in my email. I have been trying to find out where to send an email to notify about this issue and after browsing the website i have not found an email address or anyother point of contact. It would be great if i have received the coupon code.

And lastly, it is true that I am currently in Poland and have tried using skype from Poland, but my primary language is English and not sure why did i receive email in Polish and had to go above and beyond to be able to read it.

As much as i enjoy using skype the last few days and this customer care experience have been very frustrating.
Thank you

vanwetswinkel.marc commented Saturday, Jan 1

Thumbs up for Skype on this one!! They could as well have left us in the dark explaining nothing, referring to the "no guarantees"-clause in their licence agreement and go further with their business as usual, but they didn't. Congratulations Lars and everybody else at Skype.

@sangdelan, you can avoid auto updating by changing "automatic download and install" to "notification" in you Skype advanced settings. I never had any attempt to auto install a new version.

swanscombetkd commented Saturday, Jan 1

Well done for putting your hands up an being honest, Skype is a very good and very useful piece of software, more people need to subscribe to give skype the income they need to have spare capacity for these instances, we have become a nation that loves everything for free, that is of course unless its the latest gadget and it appears we will pay any price to be "cool"

simonderricutt commented Saturday, Jan 1

A good explanation - thanks. As noted by oldmrbill and others, it's refreshing to get truth and it increases my confidence in the product. On the subject of supernodes, are these normal users with extra-large machines, and do they know they are servicing others? My normal Skype use is via a Dualphone, which I suspect has no spare capacity for such services. As other (e.g. mobile phones) use Skype will the load on such supernodes increase?
As a note to melaperson, when I'm paying around 3.5 euros per month for the service a payback of 1 euro for a 24-hour hiatus seems around 9 times more generous than necessary. I wish I got the same service from, say, the airlines or my ISP.

ffboots commented Sunday, Jan 2

Some suggestions based on personal experience with the Corp of Engineers' email system in 1991:
A) Do not have supernodes auto update, let them ask the user
If a supernodes version is too old, it stops being a supernode
C) Warn Supernode users when their current version is likely to age out
D) When a user's software requests permission to do an update, it displays the estimated time the update is likely to take based in part on how long it took a nearby user
E) Users do not have to upgrade to the latest version, some recent versions that are considered rock solid can be chosen
The trick is to ensure the system can not do itself in - really hard^2 to model.
TTFN Art "Boots" Coleman

browncatdc commented Monday, Jan 3

I wasn't hugely inconvenienced by this glitch, and have for the most part been pleased with Sykpe's services. But I find their response to this problem preposterous. As with others above, I received an email of "apology" from Skype a few days ago, with a offer of a voucher redeemable for "30 minutes" of landline calls. The amount? $1.00. I would have had more respect, and been more understanding, had the company simply sent an apology and left it at that. Better nothing that this piddly, insulting sum, which - as with users above - won't cover more than a few minutes of the calls I usually use Skype to make. There are a number of ways in which Skype could have sought to achieve redress for the inconvenience caused to their customers - this was the poorest. As soon as Google Chat offers VoIP, I'm done with this company.

machone1 commented Monday, Jan 3

Thank you for the thoughtful gesture. The way Skype handled this provides a benchmark for other services companies to aspire to.

Mark.

smithhaddon123 commented Monday, Jan 3

Excellent Mr Rabbe and Mr Bates. And thanks for the voucher, very much appreciated. Excellent service. Ben Smith-Haddon

mattrissell commented Monday, Jan 3

Well articulated explanation of the outage...
Despite best efforts, outages happen to every company regardless of it's size. Honesty and transparency are the best tools to keep a happy and loyal customer base - check! You nailed it.

Happy New Year and may 2011 be outage free!

All the best,
Matt

roturco commented Monday, Jan 3

Great job at getting everyone back online, can't always expect things to be perfect. Although I do have to question...if you offer a "gratitude with a credit voucher worth a call of more than 30 minutes to a landline in some of our most popular countries, such as USA, UK, Germany, China, Japan" why did I see a charge on my account for $1 for redeeming this. You mentioned it as a CREDIT VOUCHER, right? A little confused here.

hantwister commented Wednesday, Jan 5

Curious, is there any mechanism in Skype's protocol whereby clients talking to each other know the other client's OS and version number (many protocols seem to have a stage where the software identifies itself to the other party)? In addition to checking for updates every x days, the software could also possibly know it needs to check for updates if it sees that you are actively having a call with someone else using a higher version of Skype on the same OS, for example.

macrovita commented Wednesday, Jan 5

As others, I also appreciate how Skype handled the incident. As a result of this handling, in my view Skype's image actually improves (provided it does not happen again very soon of course).

That said, as others, I also think that forcing all users to upgrade to the latest version is indeed one of the root causes of the problem. What if the failure had been on version 5.0.0.156 after it hit the majority of the users?

paulsouthernskype commented Thursday, Jan 6

Good job Lars in explaining this. Skype - a new telco - has already differentiated itself from old-telco with technology and business-model... now you're doing it in PR & communications. Congrats.

onltimo commented Thursday, Jan 6

As a business, we understand that things sometimes break. That being said, we depend on this service heavily for communications. The voucher you sent out is useless to us as there seems to be no way to apply it to a business account. If you are going to send something out, make sure its useful or it just makes your customers unhappy.

ladyditexas commented Friday, Jan 7

Only 18 comments out of the 18 million or so users? Must not have been that big of a deal. Slightly inconvenient for me, but made me appreciate you that much more really. Thank you for your excellent services and for your good communications afterward. And thank you for the gift of free minutes, which you certainly didn't have to do, and which most of us do appreciate, (except for ingrates like markskessler.) Loved reading that you're facilitating communications for Iranians like MRK. Keep up the great work.

punlman commented Thursday, Jan 13

Lars, you deserve praise for your candor in explaining what happened. It is appreciated.
I wish that Skype's non-existant customer service, and Skype's non-existant customer communication, could be just 1% as effective as this single blog post.
In general, on any other day, Skype seems to not care about its users AT ALL.
Not only do the users keep Skype afloat financially, but Skype USERS ARE the so-called "supernodes", providing the infrastructure and bandwidth that makes Skype run, LITERALLY.

Very much like Microsoft Windows, every newer version of Skype software has been FAR WORSE than the previous version. Every Skype user knows that.
There hasn't been a good solid stable reliable version of Skype since version 3.8 (and yes, everyone knows, you can Google skype 3.8 and still get it -- which I highly recommend to everyone). The Linux version works much better than any of the Windows 4.x versions. And the 5.x versions don't run AT ALL! I don't understand how you can even offer the 5.x versions to the public with a straight face.
Anyway, Skype is 99% a peer-to-peer network --- just like Bittorrent --- and we're all waiting for the day when FOSS P2P software catches up to Skype in audio quality.
When that day comes, PROPRIETARY Skype will be history.

r808moore commented Friday, Mar 4

The idea of a credit was nice. Would have been nicer if you had mentioned a deadline to take advantage of it. Sort of like a gift card that expires--except the gift card tells you how long you have to take advantage of it.

barry_harvey10 commented Saturday, Mar 5

Thanks for the detailed explaination - very refreshing to hear someone telling it how it is. This is a great service. I only have one other piece of feedback; my voucher code didn't work.